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ABSTRACT 



This study considered the possibility that different formats 
of objective test questions might differentially favor males or females and 
that males and females might respond differently to objective questions aimed 
at assessing abilities at different levels of Bloom's cognitive domain. Class 
tests were constructed on recently taught topics, with each test containing 
questions in three parallel subtests, multiple-choice, true- false, and 
matching. Each subtest had six questions, and each of the questions was 
targeted to one level of Bloom's Cognitive Domain by the test writers. 
Questions at each level were matched to the same expected difficulty level by 
the writing team using a variant of the Angoff method. This design was 
replicated in 5 schools across 4 curriculum areas with 65 male and 123 female 
students in grades 6 through 12. Results show only one significant difference 
in gender performances across the levels of Bloom's Cognitive Domain. This is 
a female advantage at the level of Analysis. A comparison of mean male and 
female scores on the three subtest formats also shows only one statistically 
significant advantage- -an advantage for females on the matching questions. 
This was found to be due to significant female advantages at the Analysis and 
Synthesis levels. The relationship and relevance of these findings is 
discussed in relation to gender differences in science and mathematics test 
performances. (SLD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Gender Differences for 6-12 th Grade Students Over Bloom’s Cognitive Domain 



vO 

oo 

r- 



VO 



s 



Tony Bastick 

University of the West Indies 



Paper presented at the Western Psychological Association 
WPA 2002 Convention 
Irvine, CA 
USA 



Publication Date: April, 2002 




lEPARTMENT OF EDUCATION 

Educational Research and Improvemem 
DNAL RESOURCES INFORMATION 
CENTER (ERIC) 

ocument has been reproduced as 
3d from the person or organization 
iting it. 

changes have been made to 

/e reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



XJ&siizk --- — 



BEST COPY AVAILABLE 



to ^» b « es 



2 



Gender differences for 6-12 th Grade Students 
Over Bloom’s Cognitive Domain. 



Tony Bastick 

University of the West Indies 



Abstract 

Differential item analysis (DIF) is routinely used to identify test items that may advantage male or 
female students. Male advantages have been found for spatial reasoning questions and female advantages 
have been found for verbal questions. Such DIF loaded test items can be discarded to produce gender 
equitable tests. However, little research seems to have compared male and female abilities at different 
levels of Bloom’s Cognitive Domain. It is therefore possible that items testing these different cognitive 
levels may differentially favour males or females, resulting in the use of gender inequitable tests. 

To test this possibility, class tests were constructed on recently taught topics. Each test contained 
1 8 questions in three parallel subtests - a Multiple-choice subtest, a True/False subtest and a subtest of 
Matching questions. Each subtest had six questions and each of the six questions was targeted to one 
level of Bloom’s Cognitive Domain by the test writers - the class teacher and two teacher trainees. 
Further, the questions at each level were matched to the same expected difficulty level by the writing 
team using a variant of the Angoff method. This design was replicated in five schools across 4 curriculum 
areas with 65 male and 123 female, 6-1 2“’ grade students. The mean scores for males and females were 
compared for each level of Bloom’s Cognitive Domain and for each subtest format. 

Results showed that there was only one statistically significant difference in gender performances 
across the levels of Bloom’s Cognitive Domain. That was a female advantage at the level of Analysis. A 
comparison of mean male and female scores on the three subtest formats also showed only one statistically 
significant advantage. That was also an advantage for females, on the Matching subtest. When this 
female advantage on Matching questions was examined more closely, it was found to be due to significant 
female advantages at the Analysis and Synthesis levels of the Matching questions, and not at the other 
levels. Overall, girls showed superior analytical ability. 

The relationship and relevance of these significant findings is discussed in relationship to gender 
differences in science and mathematics test performances. 



Introduction 

This study considered the possibility that different formats of objective test questions might differentially 
favour males or females and that males and females might respond differently to objective questions aimed at 
assessing abilities at different levels of Bloom’s cognitive domain. There is extensive research on differential 
gender responses to questions in various content areas but little research considering differential gender responses 
to the formats of objective tests at the levels of Bloom’s cognitive domain to which these questions are targeted. 

Many researchers have studied gendered content of test items to test the belief that males perform better on 
items with masculine content and ferpales perform better on items with feminine content. There is weak support 
for such simplistic content analysis of test items (Ross, 1988). Differential item functioning is routinely used to 
identify test items that are gender biased (Hamilton, 1999). These analyses are frequently performed on very large 
national samples and are able to detect very fine gender biases. For example Abigail Harris and Sydell Carlton 
(1993) used 181,228 male and 198,668 female students’ Scholastic Aptitude Test results to examine gender 
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differences related to item format and content of mathematics items. The female disadvantage on spatial items has 
been found to be most marked. In a large study (n=l 16,017) Asa Makitalo (1996) found that female performance 
on spatial items can vary substantially according to the formatting of tables and maps. Anita Wester (1995) analysed 
similar data to find if changing item format, from multiple choice to open response, would reduce these gender 
differences. She found that this did not reduce the item bias. Although gender bias has been studied in most school 
subjects, there seems to be a preponderance in the literature of differential gender studies in mathematics (Bielinski, 
& Davison, 1 998; Garner, & Engelhard, 1 999). However, there are very few studies that compare gender differences 
in objective test formats at different levels of Bloom’s cognitive domain that are not topic centred. This study 
contributes in this under-represented research area. 

Method 

The following design was replicated in five classes, each in a different school. A 20-minute class test was 
constructed on a recent topic taught by the class teacher. The test was comprised of 1 8 questions in three different 
objective item formats; being six multiple choice (MC), six True/False (T/F) and six Matching (M) questions. 
Within each item format each of the six questions was aimed at assessing a different level of Bloom’s cognitive 
domain. Each test was written by three teachers, one of whom was the class teacher who had recently taught the 
subject to be tested and two teachers in training. The three teachers adjusted the questions at the same cognitive 
level to be of the same expected difficulty. The expected difficulties were not matched between levels. So, for 
example all Knowledge questions were of the same expected level of difficulty and all Analysis questions were the 
same level of difficulty, but the Knowledge questions were not necessarily set at the same expected difficulty as the 
Analysis questions. The test formats were then randomised and the six questions within each format were also 
randomised for the class test. Figure 1 is a schematic showing the six levels of each question in the three formats 
before and after the order was randomised. A different random sequence was used to randomise the test formats 
and questions for each of the five classes. 



Figure 1 : Structure of class tests. Three objective test formats, each of six questions with each assessing 
a different level of Bloom’s cognitive domain, and all questions at the same Level having equal expected 
difficulty. 



Structure of each test 


18 Objective Questions - Questions at the same level have same expected difficulty 


Format of questions 


Six Multiple Choice Questions 
MC 


Six True/False Questions 
TF 


Six Matching Questions 
MQ 


Cognitive ability Level 


LI 


L2 


L3 


L4 


L5 


L6 


LI 


L2 


L3 


L4 


L5 


L6 


LI 


L2 


L3 


L4 


L5 


L6 


Question Number 


Q1 


Q2 


Q3 


Q4 


Q5 


Q6 


Q7 


Q8 


Q9 


Q10 


Q11 


Q12 


Q13 


Q14 


Q15 


Q16 


Q17 


Q18 



Example of random 
presentation order 


MQ 


TF 


MC 


Q16 


Q15 


Q13 


Q14 


Q17 


Q18 


Q10 


Q8 


Q9 


Q11 


Q7 


Q12 


Q15 


Q14 


Q13 


Q16 


Q17 


Q18 


Queston order on test 


T 1 


T2 


T3 


T4 


T5 


T6 


T7 


T8 


T9 


T10 


Til 


T 12 


T 13 


T 14 


T 1 5 


T 16 


T17 


R18 



Subjects 



The sample of five schools, whose principals agreed to their schools and teachers participating in this research, 
were drawn from high and low socio-economic-status populations and represents both urban and rural areas in and 
around Kingston, Jamaica, West Indies. 1 88 students, took part in this study from five classes, one in each secondary 
school. The students were 65 boys and 123 girls, aged from 10 to 16 years with a mean age of 12 years 8 months. 
Class tests were written in four topic areas. The numbers of males and females in each of the topics tested were 
Biology 1 - sexual reproduction in flowering plants (m=13, f=16, n=29), Biology 2 - Endocrine Systems (m=18, 
f=15, n=42), Physics - States of matter (m=18, f=15, n=33), English - Nouns (m=15, f=35, n=50) and Social 
Studies - The Family and The Peer Group (m=l 1, f=23, n=34). 
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Analysis and results 

The responses for each test were scored by the three teachers and entered into SPSS for global analysis. The 
separate performances of boys and girls were then compared on the three matched subtests. These mean percentage 
scores are illustrated in Figure 2 and tabulated in Table 1 with the mean scores for each subtest and the significance 
of the gender differences. 



Figure 2: Comparative gender performance on three different test formats matched for expected difficulty 




Test Format 



Key: MC - Multiple Choice format. TF - True/False format. M - Matching format 



Table 1 : Mean Percent Scores for each of three item formats of equivalent expected difficulty by sex of 
respondents 



Scores as 


Objective Test Formats 


% of each 
item format 


Multiple 

Choice 


True 

False 


Matching 


Sex 


Males 


61 


59 


51 


Females 


66 


61 


68 


Significance 


P=0.106 


P=0.636 


P=0.000 



The most significant sex difference was the superior performance of the girls on the Matching Subtests. The 
Matching subtests, like the other subtests, were composed of one question at each level of Bloom’s cognitive 
domain. The scores of boys and girls were then compared at each domain level for the Matching subtests, across all 
school topics tested, in order to isolate the levels of abilities contributing to female advantage on Matching subtests. 
Figure 3 illustrates these sex differences in performance at each level and Table 2 lists the mean scores and the 



sianificances of these gender differences. 
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Figure 3 : Sex differences at each Level of Bloom’s cognitive domain for Matching format items 
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Matching format questions at each Level of Bloom's Cognitive 

Table 2: Significant differences in scores of boys and girls at each Level of Bloom’s cognitive domain for 
Matching format items 



Cognitive Levels 
Matching Format only 
Question # 


LI L2 

Knowledge Comprehension 
Q13 Q14 


L3 

Application 

Q15 


L4 

Analysis 

Q16 


L5 

Synthesis 

Q17 


L6 

Evaluation 

Q18 


% Score at each Level 














Sex 














Males: (n=65) 


52 


57 


45 


43 


38 


68 


Females: (n=1 23) 


63 


66 


59 


63 


56 


72 


Significance 


P=0.174 


P=0.231 


P=0.054 


P=0.008 


P=0.024 


P=0.056 


For all: (n= 1 88) 


59 


63 


54 


56 


50 


71 



We should not pay particular attention to the relative scores of different cognitive levels, i.e. that Q18 has a 
higher mean score than Q 1 7 say, because different levels were not pre-matched for expected difficulty. However, 
we do note the significant gender differences (p<0.05) in Matching format items at the levels of Analysis (Q 16) 
and Synthesis (Q1 7). 



It is again noted that questions were not designed to be equivalent in expected difficulty across different 
Levels of the Cognitive Domain. For example, questions at Level 1 were not equated in expected difficulty with 
questions at Level 2, or any other level. Hence, the expected difficulties could not be compared across levels in this 
research. However, a question at any Level was of the same expected difficulty for boys and girls. Hence, the we 
can compare the resulting expected difficulties for boys and girls within each level to find if boys or girls are 
advantaged at each level of Bloom’s Cognitive Domain. 
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Sex differences of scores at each Level, summed across all three objective item formats, are illustrated in 
Figure 4 with the total scores and significances of the sex differences tabulated in Table 3. 

Figure 4: Sex differences in scores summed across all three objective item formats at each Level of 
Bloom’s Cognitive Domain 




Each Level of Bloom's Cognitive Domain for all three objective item formats 



Table 3: Significances of sex differences in scores summed across all three objective item formats at 
each Level of Bloom’s Cognitive Domain 

Question # Q1, Q7, Q13 Q2, Q8, Q14 Q3, Q9, Q15 Q4, Q10, Q16 Q5, Q11, Q17 Q6, Q12, Q18 

All Recognition Formats Knowledge Comprehension Application Analysis Synthesis Evaluation 

Levels LI L2 L3 L4 L5 L6 



% Score at each Level 
Sex 



Males: (n=65) 


67 


61 


48 


53 


56 


58 


Females: (n=123) 


67 


63 


55 


64 


59 


66 


Significance 


P=0.91 


P=0.702 


P=0.149 


P=0.012 


P=0.48 


P=0.077 


For all: (n=188) 


67 


62 


52 


60 


58 


63 



It will be noticed from Figure 4 that boys and girls scored virtually the same only at the lowest Level of 
‘Knowledge’. At all other Levels of objective formats girls scored more than boys. Table 3 shows particularly that 
girls outperformed boys significantly (p<0.05) in Analysis (Level 4). 

Discussion 

This research has explored differences in the way 10 to 11 year old boys and girls respond to three objective 
test formats at the six levels of Bloom’s cognitive domain. It is particularly, noteworthy that girls outperformed 
boys on all three test formats and at every cognitive level. They out-performed boys significantly in the matching 
format and within the matching format, they out-performed boys significantly at the cognitive levels of synthesis 
and Analysis. When we compared the mean responses of boys and girls for all three formats at each level, we 
noticed that girls out-performed boys significantly in analytical ability. 



GetiWKK^'erences over Bloom s Cognitive Domain: 6:6 

It is interesting to note that in subject areas that are assumed to require higher analytical ability, such as 
physics and mathematics, the literature has shown boys are usually in advance of girls, although the gap is narrowing. 
This study has shown that, at least in this sample of 188 Jamaican children, that it is not necessarily a lack of 
analytical ability that might be responsible for the gap. This implies that researchers have been correct in looking 
to social rather than cognitive areas for an explanation of female under performance in these more analytical 
subjects. Hence, these results support research into socio-cultural factors that might result in gender bias on test 
performance. 
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